First-order selection parameters

The first set of parameters affects which context words will be selected from the context of each token for the first order vectors. In the current models, the main distinction is made by BASE: between bag-of-words (BOW) based and dependency-based models (LEMMAPATH and LEMMAREL). The former are further split by window size, part-of-speech filters and whether sentence boundaries are respected.

FOC-WIN (first order window)

In BOW-based models, a symmetric window of 3, 5 or 10 tokens to each side of the target was used.

In dependency models, no restrictions to window size were imposed.

FOC-POS (first order part-of-speech)

In BOW-based models, a restriction can be placed to only select (common) nouns, adjectives, verbs and adverbs (lex) in the surroundings of the token. If no restriction is placed, the value of this parameter is all.

In dependency models, no restrictions on part-of-speech were imposed, although LEMMAREL implies them.

BOUNDARIES

In BOW-based models, yes or bound indicates that only features in the same sentence as the target were included, while no or nobound ignores sentence delimiters.

Dependency models are, of course, always yes.

LEMMAPATH

This set of dependency-based models selects the features that enter a syntactic relation with the target with a maximum number of steps. Features more than 3 steps away from the target were always excluded.¹

A one-step dependency path is either the head of the target or its direct dependent. Such features are included by both selection2 and selection3 and receive a weight of 1 in weight.

A two-step dependency path is either the head of the head of the target, the dependent of its dependent, or its sibling. Such features are included by both selection2 and selection3 and receive a weight of 2/3 in weight.

A three-step dependency path is either the head of the head of the head of the target, the sibiling of the head of its head, the dependent of the dependent of its dependent or the dependent of a sibling. An example of the last path is the subject of a passive construction with a modal, where the target is the verb in participium. Such features are included in selection3 but excluded from selection2 and receive a weight of 1/3 in weight.

LEMMAREL

This set of dependency-based models selects the features that enter in a certain syntactic relation with the target. The paths were based on the top paths selected as cues (see Annotation) and, while they account for conjuncts and some modals, they don’t include long chains of them.

group1 includes direct objects, active and passive subjects (with up to two modals for the active one), reflexive complement and prepositions depending directly on the target.

group2 expands group1 with conjuncts of the verb, complementizers, nouns depending through a preposition and verbal complements or elements of which the target is a verbal complement.

PPMI weighting

The PPMI parameter is taken outside the set of first-order parameters because it can both filter out first-order features and reshape their vector representations. In truth, the choice of positive pointwise mutual information (PPMI) over other weighting mechanisms, as well as setting a threshold or not, is already a parameter setting, which in these circumstances is set to PPMI and a threshold of 0. In all cases, the PPMI was calculated based on a 4-4 window (that could also be a variable parameter).

This parameter can take three values. selection and weight mean that only the first-order features with a PPMI > 0 with the target type are selected, and the rest discarded, while no does not apply the filter. The difference between selection and weight is that the former only uses the value to filter the context features, while the latter also weighs their vectors with that value.

Second-order selection

The selection of second-order features influences the shape of the vectors: how the selected first-order features are represented. While the frequency transformation and the window on which such values were computed could be varied, they were set to fixed values, namely PPMI and 4-4 respectively. The parameters that were varied across, although we don’t expect drastic differences between the models, are vector length and part-of-speech.

SOC-POS (second order part-of-speech)

This parameter can take two values: nav and all. In the former case, a selection of 13771 lect-neutral nouns, adjectives and verbs made by Stefano is taken as the set of possible second-order features. In the latter, all lemmas with frequency above 227 and any part-of-speech are considered.

LENGTH

Vector length is the number of second-order features and therefore the dimensionality of the matrices on which the distance matrices are based, although the amount is not all that changes. It is applied after filtering by part-of-speech.

We have selected two values: 5000 and FOC. The former includes the 5000 most frequent elements of the possible features, while the latter takes the intersection between the possible second-order-features and the first-order-features, regardless of frequency. With SOC-POS:all, FOC will include all first-order features of that model, while with SOC-POS:nav, only those included in Stefano’s selection.

The actual number of dimensions resulting from FOC depends on the strictness of the first order filter. This information can be found on the plots that, for each lemma, show how many first order context words are left after each combination of first order filters.

Based on visual examination

As a first step in the analysis, the level1 clouds of all the verbs where examined to see which parameters grouped models together in its nMDS reduction. For each level, two kinds of distance matrices were created from a procrustes analysis of the token-level matrices: one with the original data, and one with a log transformation (log(1 + log(rank)), rank being the row-wise ranks of the original distance matrix). While this gave slightly different structures for each lemma (except for haken, where they were identical), which parameters were strongest stayed usually the same. The discussion below pertains to the procrustes based on the transformed matrices, but unless specified, it also applies to the non transformed ones.

There are five main situations that could be identified in the different lemmas and that seemed relevant. The groups, diagrammed in Figure 1, are:

• FP (middle rows): There is a main split based on FOC-POS, with FOC-POS:lex separated from the rest and FOC-POS:all between it and the dependency-based models (or at least closer to the dependency-based models).
• FW (bottom rows): There is a sequential split based on FOC-WIN, with FOC-WIN:5 between FOC-WIN:3 FOC-WIN:10. In one case the dependency-based models are on the other side of FOC-WIN:3 (herstructureren), in another they are on the side of FOC-WIN:10 (heffen), but in the rest the BASE-division is transversal to the FOC-WIN-division.
• P (middle columns): There is some sort of organization based on PPMI. It isn’t very strict because, except for herhalen, the distinction between models with different PPMI is not so clear.
• LSP (right columns): The models with LENGTH:5000 + SOC-POS:all are grouped together and separated from the rest. This can be crosscutting one of the other divisions or even as the strongest split.
• LR1 (black text instead of gray): The models with LEMMAREL:group1 are clustered together independently from the rest of the divisions (sometimes with the LSP group).

Figure 1. Verbs organized by most important parameters.

Based on v-measures

In order to objectify the previous description of the effect of parameters, v-measures were calculated to estimate the variance explained by each parameter. Concretely, for each lemma a hierarchical cluster analysis was performed, and its results were evaluated against the values of each parameter. High values mean that the clustering solution agrees with grouping based on a certain parameter. A number of clustering methods were used (“ward.D”, “single”, “complete”, “average”, “mcquitty”, and “ward.D2” in the hclust() function)²; instead of looking for “high” values (which varied greatly across lemmas, parameters and clustering methods), we extracted the parameters with the highest 4 (out of 9) values based on at least 5 methods. While the quality of the match between the clustering solution and the parameter may vary, the idea is to extract the most important ones.

This workflow was applied both to the nMDS solutions and to the procrustes distance matrices themselves. The most important parameters are FOC-WIN, FOC-POS, BASE + BOUNDARIES (division between BOW-based with boundaries, BOW-based without boundaries and dependency-based) and SOC-VECTOR (interaction between LENGTH and SOC-POS, quite close to the LSP group above). The comparison between the resulting groupings is shown in Table 1.

A large number of groupings (in bold in the table) are “confirmed” by both v-measure calculations. However, this relationship should not be taken at face value. First, it must be noted that only the ranking of the v-measure values was taken into account, so the variance explained by each parameter might still be low; in very few cases go the v-measure values above 0.1. Moreover, the patterns caught by the visual inspection don’t match perfectly what the v-measures assess.

• When it comes to FOC-WIN, v-measures will expect to match four clusters to the three FOC-WIN values and the dependency-based models. FW is more abstract and more specific at the same time: more abstract because it can allow for different degrees of separability, and for FOC-WIN and BASE to use different dimensions (i.e. splitting the plot orthogonally); more specific because it requires the FOC-WIN organization to be sequential, with FOC-WIN:5 clearly between the other two.
• In the case of FOC-POS, v-measures will expect to match three clusters to the two FOC-POS values and the dependency-based models, while FP rather captures a contrast between FOC-POS:lex on the one hand and FOC-POS:all|NA on the other, allowing for different degrees of overlap between the latter two. A better matching division was also tested with the v-measures, which contributed one lemma in each case (based on 2D or procrustes).
• V-measures will also assess SOC-VECTOR as to how well the four main clusters match its four values, while LSP only requires for LENGTH:5000 + SOC-POS:all to be separated from the rest. A better matching division was also tested with the v-measures but it was not high enough in the rankings.

Visual comparison

In order to compare models along a defined set of parameters, the following groups of clouds were looked at side by side:

• Three groups of 8 models with BOUNDARIES:yes + FOC-POS:lex|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-WIN and BASE but also assess the effect of PPMI.
• Three groups of 8 models with BOUNDARIES:yes + FOC-POS:lex|NA + LENGTH:5000 + SOC-POS:all, each with a different PPMI, in order to compare across different FOC-WIN and BASE but also assess the effect of PPMI and LENGTH+SOC-POS.
• Three groups of 6 models with BOUNDARIES:yes + BASE:BOW + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-WIN and FOC-POS within the BOW-based models but also assess the effect of PPMI.
• One group of 9 models with LEMMAREL:group1 | LEMMAPATH + LENGTH:FOC + SOC-POS:nav to compare across different dependency-based models but also assess the effect of PPMI.

The differences between BOUNDARIES was not explored because the models seem to be very close to each other. For the LENGTH:FOC + SOC-POS:nav models, distances rarely go above 0.3, while for LENGTH:5000 + SOC-POS:all they go up to around 0.5 (which is still quite low). Values are higher for pairs of values with different BASE and for PPMI:no.

The heffen clouds tend to have very good separability, quite homogeneous in terms of senses but not complete. In the first set, even the nMDS solutions show distinct clusters, with greater distance in PPMI:weight and smaller in PPMI:no. One of the clusters corresponds to heffen_2 and the other two to heffen_1: one for the collocation with glas and one for the rest (mostly collocation with hand+hemel, arm and vinger). In models with PPMI:selection|weight, some smaller clusters for the co-occurrence with tol and accijns (heffen_2) are visible, especially in t-SNE models with low perplexity, as well as two entrée (heffen_2) tokens together and far from the rest of heffen_2.

FOC-WIN:10 + PPMI:no seems to decrease the distance between the heffen_1 subgroups (at least in t-SNE with perplexity 20) and both that combination and LEMMAPATH + PPMI:no pull the glas cluster towards heffen_2 because of the co-occurrence with worden.

These very evident clusters in LENGTH:FOC + SOC-POS:nav are not as clear in the nMDS solutions of LENGTH:5000 + SOC-POS:all unless PPMI:weight (which leads to two separate glas clusters in t-SNE with perplexity 10). While the senses are relatively grouped together, they are not as neat.

The rest of the groups don’t add significant insights.

Clustering and separability measures

One other way of selecting the “best” models, or of quantifying their quality, is to use separability indices (from the semvar package). The four main indices (“DR”, “SIL”, “SCP” with k=10, and “kNN” with k=10) have been computed and their values (global quality, quality by class and mean class quality) have been ranked. The 8 models whose values lie in the top 10% for all ranks are listed below. They are mostly t-SNE solutions with perplexity 50, LENGTH:FOC + FOC-WIN:10 + FOC-POS:lex + PPMI:selection|weight.

• BOWnobound10lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.50)
• BOWbound10lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.50)
• BOWbound10lex.PPMIselection.LENGTHFOC.SOCPOSnav (tsne.30)
• BOWnobound10lex.PPMIweight.LENGTHFOC.SOCPOSall (tsne.50)
• BOWbound10lex.PPMIselection.LENGTHFOC.SOCPOSnav (tsne.50)
• BOWnobound10all.PPMIweight.LENGTHFOC.SOCPOSall (tsne.50)
• BOWbound10lex.PPMIweight.LENGTHFOC.SOCPOSall (tsne.50)
• BOWbound10lex.PPMIselection.LENGTHFOC.SOCPOSall (tsne.50)

Of these methods, the ones with highest ranking v-measures across different methods are BOWbound10lex.PPMIweight.LENGTHFOC.SOCPOSall (tsne.50) and BOWnobound10all.PPMIweight.LENGTHFOC.SOCPOSall (tsne.50), illustrated in Figure 5.

Figure 5. Best models of ‘heffen’ according to separability indices and v-measures.

The actual separability indices for these two best models is shown in Table 2.

More on separability indices

While the “best” models were chosen based on a ranking, the range of the values is very different across lemmas. The curve of the different values by rank for heffen is shown in Figure 6.

The highest values of the local indices (“kNN” and “SCP”) are very high and decrease very slowly, while the global ones decrease a bit faster at the beginning and then quite slow; the maximum Silhouette (“SIL”) values are a bit over 0.6, which is quite reasonable. Furthermore, the values are very similar for all senses and globally.

Figure 6. Separability indices of ‘heffen’ by rank for different measures and levels.

Figure 7. Separability indices of ‘heffen’ by harmonic mean of ranks for different measures and levels.

Visual comparison

In order to compare models along a defined set of parameters, the following groups of clouds were looked at side by side:

• Three groups of 8 models with BOUNDARIES:yes + FOC-POS:lex|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-WIN and BASE but also assess the effect of PPMI.
• Three groups of 8 models with BOUNDARIES:yes + FOC-POS:lex|NA + LENGTH:5000 + SOC-POS:all, each with a different PPMI, in order to compare across different FOC-WIN and BASE but also assess the effect of PPMI and LENGTH+SOC-POS.
• Three groups of 6 models with BOUNDARIES:yes + BASE:BOW + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-WIN and FOC-POS within the BOW-based models but also assess the effect of PPMI.
• One group of 9 models with LEMMAREL:group1 | LEMMAPATH + LENGTH:FOC + SOC-POS:nav to compare across different dependency-based models but also assess the effect of PPMI.

The differences between BOUNDARIES was not explored because the models seem to be very close to each other. Distances only go below 0.2 for dependency-based models with the same template or for BOW-based models with the same FOC-WIN and PPMI:selection|weight. They can go up to 0.8 or 0.9, especially between models where one is FOC-WIN:3 or LEMMAREL.

The main characteristic of the harden clouds is the presence of a very large and very idiomatic sense, namely harden_5 (‘niet te harden’). It makes for a dense core in the nMDS solutions and for a very strong, well formed cluster in LEMMAREL:group1 t-SNE solutions, even with low perplexity, and with any SOC-VECTOR. In other models, this sense is split by other means: FOC-WIN:3 and LEMMAPATH:selection2 distinguish between the presence and absence of meer (“niet [meer] te harden”), while the rest group tokens based on popular objects, mainly pijn and stank. The latter is more clear in LENGTH:FOC + SOC-POS:nav, but always regardless of PPMI.

There is no clear structure for the rest of the senses, except for a relatively low overlap with harden_5 in t-SNE solutions.

Clustering and separability measures

One other way of selecting the “best” models, or of quantifying their quality, is to use separability indices (from the semvar package). The four main indices (“DR”, “SIL”, “SCP” with k=10, and “kNN” with k=10) have been computed and their values (global quality, quality by class⁴ and mean class quality) have been ranked. The top 10% for harmonic mean of ranks includes 33 solutions; there is no overlap between the top 10% of the different measures.

Of these methods, the ones with highest ranking v-measures across different methods are BOWnobound3all.PPMIselection.LENGTH5000.SOCPOSnav (mds) and LEMMAPATHselection3.PPMIweight.LENGTH5000.SOCPOSnav (tsne.10), illustrated in Figure 11.

Figure 11. Best models of ‘harden’ according to separability indices and v-measures.

The actual separability indices for these two best models is shown in Table 3.

More on separability indices

While the “best” models were chosen based on a ranking, the range of the values is very different across lemmas. The curve of the different values by rank for harden is shown in Figure 12.

The local values are relatively high, lower for harden_3 than for harden_5, and decrease slowly, while the global values decrease much faster (especially harden_5, which starts quite high).

Figure 12. Separability indices of ‘harden’ by rank for different measures and levels.

Figure 13. Separability indices of ‘harden’ by harmonic mean of ranks for different measures and levels.

Visual comparison

In order to compare models along a defined set of parameters, the following groups of clouds were looked at side by side:

• Three groups of 8 models with BOUNDARIES:yes + FOC-POS:lex|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-WIN and BASE but also assess the effect of PPMI.

Distances between these models range between 0.2 and 0.8, in the upper half for those with different BASE and lowest third for LEMMAPATH pairs of models, with slightly higher values between PPMI:no models than between PPMI:weight models.

While the overlap between diskwalificeren_2 (the “sports” sense) and the rest varies depending on the model and solution, diskwalificeren_3 (reflexive, semantically similar to diskwalificeren_1) consistently appears included in the diskwalificeren_1 area, even if tightly clustered (sometimes split in two based on the opposition between zich and zichzelf). In a number of models, diskwalificeren_2 has two main clouds: one in which vals, start and meter tend to co-occur, and one in which it doesn’t. A frequent collocate is worden, which can pull tokens from both main senses together.

Clustering and separability measures

One other way of selecting the “best” models, or of quantifying their quality, is to use separability indices (from the semvar package). The four main indices (“DR”, “SIL”, “SCP” with k=10, and “kNN” with k=10) have been computed and their values (global quality, quality by class and mean class quality) have been ranked. The top 10% for harmonic mean of ranks includes 27 solutions; there is no overlap between the top 10% of the different measures.

Of these methods, the ones with highest ranking v-measures across different methods are LEMMAPATHweight.PPMIselection.LENGTHFOC.SOCPOSall (tsne.30) and LEMMAPATHweight.PPMIselection.LENGTHFOC.SOCPOSnav (tsne.50), illustrated in Figure 17.

Figure 17. Best models of ‘diskwalificeren’ according to separability indices and v-measures.

The actual separability indices for these two best models is shown in Table 4.

More on separability indices

While the “best” models were chosen based on a ranking, the range of the values is very different across lemmas. The curve of the different values by rank for diskwalificeren is shown in Figure 18.

The highest local values are quite high for diskwalificeren_2, a bit lower (around 0.75) for diskwalificeren_" and lowest for diskwalificeren_1, but they decrease quite rapidly and drastically for diskwalificeren_3. The silhouete is lower but has a similar shape, decreasing slowly for all but diskwalificeren_3.

Figure 18. Separability indices of ‘diskwalificeren’ by rank for different measures and levels.

Figure 19. Separability indices of ‘diskwalificeren’ by harmonic mean of ranks for different measures and levels.

Visual comparison

In order to compare models along a defined set of parameters, the following groups of clouds were looked at side by side:

• Three groups of 8 models with BOUNDARIES:yes + FOC-POS:lex|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-WIN and BASE but also assess the effect of PPMI.
• Three groups of 9 models with BOUNDARIES:yes + FOC-WIN:3|10|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-POS and BASE but also assess the effect of PPMI.

Distances in the first three groups span between 0.1 and 0.8, with highest values for LEMMAREL pairs if PPMI:weight (around .5 otherwise), lowest for BOW-based and LEMMAPATH pairs and the upper half for models with different BASE, with LEMMAPATH closer to BOW-based than LEMMAREL. When FOC-POS is considered, in the second set, the lowest distances are between FOC-POS:all and LEMMAPATH models, followed by BOW-based models and FOC-POS:lex with LEMMAPATH and finally pairs with LEMMAREL.

LEMMAREL:group1 offers consistently the worse models, so they won’t be discussed. Suffice to say that whatever clusters are found in the other models, are much harder to identify in LEMMAREL:group1.

In all nMDS solutions in both sets with the notable exception of LEMMAREL:group1 (and less defined in FOC-WIN:3 and LEMMAREL:group2), there is a clear small, dense cloud with most of the haken_3 (“make someone trip”) tokens, clustered by worden and strasfschopgebied in all of them, strafschop in FOC-WIN:10 and in, en and maar in LEMMAPATH. There is little overlap between the senses, except for haken_2 (intransitive “hook”), which does overlap with haken_1 (transitive “hook”), haken_4 (figurative, intransitive “hook”) and haken_6 (with naar, “want, seek”), which on their part don’t overlap much with each other.

Among the t-SNE solutions, perplexity 20 and 30 are very similar and exhibit tighter clusters than perplexity 10, while perplexity 50 is not useful anymore. The haken_3 cluster is very clear, especially in dependency-based models, and in FOC-WIN:3 + PPMI:selection|weight it’s split between strafschop and strafschopgebied. haken_5 (crochet) and haken_6 (with naar) are particularly well separated in dependency-based models. With perplexity 20, another cluster with mostly haken_2 tokens emerges, characterized by the occurrence of in and elkaar.

All these clusters are well defined in LEMMAPATH models; haken_3 is a bit weaker in LEMMAREL:group2 and FOC-WIN:3, haken_5 prefers FOC-WIN:10 + FOC-POS:lex to other BOW-based models, and haken_6 and ‘in elkaar’, FOC-POS:all. They don’t seem to mind too much about PPMI.

Clustering and separability measures

One other way of selecting the “best” models, or of quantifying their quality, is to use separability indices (from the semvar package). The four main indices (“DR”, “SIL”, “SCP” with k=10, and “kNN” with k=10) have been computed and their values (global quality, quality by class and mean class quality) have been ranked. The top 10% for harmonic mean of ranks includes 53 solutions; there is no overlap between the top 10% of the different measures –actually, there is not even overlap between the top 10% of all values of “kNN” or “SIL”.

Of these methods, the ones with highest ranking v-measures across different methods are LEMMAPATHweight.PPMIselection.LENGTHFOC.SOCPOSall (tsne.10) and its tsne.30 counterpart, illustrated in Figure 23.

Figure 23. Best models of ‘haken’ according to separability indices and v-measures.

The actual separability indices for these two best models is shown in Table 5.

More on separability indices

While the “best” models were chosen based on a ranking, the range of the values is very different across lemmas. The curve of the different values by rank for haken is shown in Figure 24. DR values were ignored for 1 solution because they were too high –larger than 10– in some levels (classqual_haken_6), up to 10.46.

The higest local values are very high for haken_3 and haken_6, but the latter decreases drastically, while the rest are a bit slower, with values mostly above 0.5 for haken_2 and haken_3 and lower for the rest. The silhouet decreases just as softly (and more drastically for haken_5 and haken_6), with maximum values barely over 0.5 and mostly negative values.

Figure 24. Separability indices of ‘haken’ by rank for different measures and levels.

Figure 25. Separability indices of ‘haken’ by harmonic mean of ranks for different measures and levels.

Visual comparison

In order to compare models along a defined set of parameters, the following groups of clouds were looked at side by side:

• Three groups of 8 models with BOUNDARIES:yes + FOC-POS:lex|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-WIN and BASE but also assess the effect of PPMI.
• Three groups of 9 models with BOUNDARIES:yes + FOC-WIN:3|10|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-POS and BASE but also assess the effect of PPMI.

The lowest distances, normally below 0.25, are between dependency-based models; it’s a bit higher between dependency-based and FOC-POS:all models, then FOC-POS:lex pairs, followed by BOW-based models with different FOC-POS and finally pairs with one dependency-based model and one FOC-POS:lex, reaching 0.8 or even higher.

In nMDS solutions, there’s a dense herinneren_2 core with a fuzzy halo and overlap between senses in the FOC-POS:lex models and better separation between them in FOC-POS:all and dependency-based models. In PPMI:weight, three clusters emerge: one for third person herinneren_2 (collocation with ‘zich’), one for first person herinneren_2 (collocation with ‘me’ and ‘ik’) and one for ‘eraan’ in herinneren_1. These clusters are most clear in FOC-WIN:10 + FOC-POS:all and LEMMAREL, and still clear but less separated in LEMMAPATH and FOC-WIN:3 + FOC-POS:all.

In t-SNE solutions, perplexity of 20 strengthens the clusters in the dependency-based models but does not improve the ones in the BOW-based models; higher perplexity does not help either. In FOC-POS:lex + PPMI:weight there can be some grouping but it’s based on the occurrence of ‘nog’, ‘niet’, ‘kunnen’ and ‘niets’. In all dependency-based and FOC-POS:all + PPMI:weight models, the two main senses are clearly separated: two groups for first and third person herinneren_2 (more separated if PPMI:weight, closer together otherwise), a small ‘eraan’ cluster and one for the rest of herinneren_1.

Clustering and separability measures

One other way of selecting the “best” models, or of quantifying their quality, is to use separability indices (from the semvar package). The four main indices (“DR”, “SIL”, “SCP” with k=10, and “kNN” with k=10) have been computed and their values (global quality, quality by class and mean class quality) have been ranked. The top 10% for all ranks includes 33 solutions.

Of these methods, the ones with highest ranking v-measures across different methods are LEMMAREL2.PPMIselection.LENGTH5000.SOCPOSnav (tsne.50) and LEMMAREL1.PPMIselection.LENGTH5000.SOCPOSnav (tsne.50), illustrated in Figure 29.

Figure 29. Best models of ‘herinneren’ according to separability indices and v-measures.

The actual separability indices for these two best models is shown in Table 6.

More on separability indices

While the “best” models were chosen based on a ranking, the range of the values is very different across lemmas. The curve of the different values by rank for herinneren is shown in Figure 30.

The highest local values are very high, decreasing slowly, and staying quite high for herinneren_2; the silhouete, instead, starts barely around 0.5 and decreases faster, although only the worst values go below 0.

Figure 30. Separability indices of ‘herinneren’ by rank for different measures and levels.

Figure 31. Separability indices of ‘herinneren’ by harmonic mean of ranks for different measures and levels.

Visual comparison

In order to compare models along a defined set of parameters, the following groups of clouds were looked at side by side:

• Three groups of 9 models with BOUNDARIES:yes + FOC-WIN:3|10|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-POS and BASE but also assess the effect of PPMI.
• Three groups of 8 models with BOUNDARIES:yes + FOC-WIN:10, each with a different PPMI, in order to compare across different SOC-POS, LENGTH and FOC-POS but also assess the effect of PPMI.

Distances span between 0.1 and 0.8, with very high values for pairs where one model is FOC-POS:lex and the other one is not, very low for pairs of dependency-based models, and in between for the rest; in the lower half when PPMI:weight and going higher when PPMI:no. Distances between LENGTH:FOC models and between SOC-POS:nav models is lower, but neglectable if there is a difference in FOC-POS.

Both in nMDS and t-SNE solutions the two main senses overlap completely. The only clusters to be found (in lower perplexities, 10 and 20) are one of ‘ik’+‘het’ (mostly haten_2) and one for ‘worden’. The former is mostly visible in LEMMAREL but also in (FOC-POS:all | LEMMAPATH) + PPMI:weight with perplexity 10, and the latter only appears in FOC-WIN:3 + FOC-POS:lex + PPMI:weight. In perplexity 20, LEMMAPATH models also seem to split the tokens according to the presence or absence of ‘en’ and ‘die’.

Clustering and separability measures

One other way of selecting the “best” models, or of quantifying their quality, is to use separability indices (from the semvar package). The four main indices (“DR”, “SIL”, “SCP” with k=10, and “kNN” with k=10) have been computed and their values (global quality, quality by class and mean class quality) have been ranked. The top 10% for harmonic mean of ranks includes only 11 solutions; there is no overlap between the top 10% of the different measures –actually, there is only overlap between the top 10% of all values of “kNN”.

• LEMMAPATHselection2.PPMIweight.LENGTH5000.SOCPOSnav (tsne.30)
• BOWbound5all.PPMIweight.LENGTHFOC.SOCPOSall (tsne.50)
• LEMMAREL1.PPMIweight.LENGTH5000.SOCPOSnav (tsne.20)
• BOWnobound5all.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.30)
• BOWbound5all.PPMIweight.LENGTHFOC.SOCPOSall (tsne.30)
• LEMMAREL2.PPMIno.LENGTHFOC.SOCPOSnav (tsne.50)
• LEMMAREL2.PPMIno.LENGTHFOC.SOCPOSnav (tsne.30)
• LEMMAREL2.PPMIno.LENGTHFOC.SOCPOSall (tsne.30)
• LEMMAREL2.PPMIselection.LENGTHFOC.SOCPOSnav (tsne.50)
• LEMMAPATHweight.PPMIselection.LENGTHFOC.SOCPOSnav (tsne.20)
• BOWnobound3all.PPMIno.LENGTHFOC.SOCPOSnav (tsne.20)

Of these methods, the ones with highest ranking v-measures across different methods are LEMMAREL2.PPMIno.LENGTHFOC.SOCPOSnav (tsne.50) and LEMMAREL1.PPMIweight.LENGTH5000.SOCPOSnav (tsne.20), illustrated in Figure 35.

Figure 35. Best models of ‘haten’ according to separability indices and v-measures.

The actual separability indices for these two best models is shown in Table 7.

More on separability indices

While the “best” models were chosen based on a ranking, the range of the values is very different across lemmas. The curve of the different values by rank for haten is shown in Figure 36.

The local values start relatively low, barely above 0.6 for haten_1 and haten_2 and around 0.2 or lower for both, but don’t decrease that much. The third, least frequent reading starts instead with the highest global values (quite low anyways), decreasing very fast.

Figure 36. Separability indices of ‘haten’ by rank for different measures and levels.

Figure 37. Separability indices of ‘haten’ by harmonic mean of ranks for different measures and levels.

Visual comparison

In order to compare models along a defined set of parameters, the following groups of clouds were looked at side by side:

• Three groups of 9 models with BOUNDARIES:yes + FOC-WIN:3|10|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-POS and BASE but also assess the effect of PPMI.
• Three groups of 9 models with BOUNDARIES:yes + FOC-WIN:3|10|NA + LENGTH:5000 + SOC-POS:all, each with a different PPMI, in order to compare across different FOC-POS and BASE but also assess the effect of PPMI and SOC-VECTOR.

Distances are normally around 0.7 or 0.8 for pairs with one FOC-POS:lex model, lower than 0.2 for LEMMAPATH pairs and in between for the rest; pairs with one LEMMAPATH model are more similar than those with a LEMMAREL model, and similarity between dependency-based models increases with PPMI:no and decreases with PPMI:weight.

In nMDS solutions, helpen_3 (inanimate, intransitive), helpen_5 (with aan) and om zeep helpen are relatively well grouped and don’t overlap with each other, unless PPMI:no; the rest of the senses do overlap. However, the models in which these senser are grouped are not always the same. In the first set of models, helpen_3 is quite clear in all but LEMMAREL:group1, and the other two in all but FOC-POS:lex; in the second set, intsead, the former is more or less clear in FOC-WIN:3, helpen_5 in (LEMMAREL:group1 | FOC-WIN:10 + FOC-POS:all) + PPMI:weight and LEMMAPATH + PPMI:selection, and om zeep helpen in LEMMAPATH | LEMMAREL:group2 | FOC-POS:all.

In t-SNE solutions, perplexity 20 seems to enhance the weak clusters found in lower perplexity, but higher values don’t improve the models: only helpen_5 is visible, and barely, in LEMMAREL:group1 + PPMI:selection|weight models of the first set; in the second it’s much less clear. The three forementioned clusters are still present, especially in PPMI:weight and with perplexity 10, and never in FOC-POS:lex models (where a cluster for "niet’+‘kunnen’ can be found instead, if FOC-WIN:3). helpen_3 is grouped by the co-occurrence of ‘niet’+‘het’ in all but LEMMAREL; helpen_5 is best clustered in LEMMAREL and split in two in the rest; om zeep helpen in all but LEMMAREL:group1. PPMI:weight also pulls other sets of tokens together, regardless of their sense tags: occurrences with ‘bij’, ‘vooruit’ and ‘bovenop’, especially in dependency-based models.

Clustering and separability measures

One other way of selecting the “best” models, or of quantifying their quality, is to use separability indices (from the semvar package). The four main indices (“DR”, “SIL”, “SCP” with k=10, and “kNN” with k=10) have been computed and their values (global quality, quality by class and mean class quality) have been ranked. The top 10% for harmonic mean of ranks includes only 39 solutions; there is no overlap between the top 10% of the different measures –actually, there is no overlap between the top 10% of all values of any measure.

Of these methods, the ones with highest ranking v-measures across different methods are LEMMAREL2.PPMIselection.LENGTH5000.SOCPOSnav (tsne.20) and LEMMAREL2.PPMIno.LENGTH5000.SOCPOSnav (tsne.20), illustrated in Figure 41.

Figure 41. Best models of ‘helpen’ according to separability indices and v-measures.

The actual separability indices for these two best models is shown in Table 8.

More on separability indices

While the “best” models were chosen based on a ranking, the range of the values is very different across lemmas. The curve of the different values by rank for helpen is shown in Figure 42.

Figure 42. Separability indices of ‘helpen’ by rank for different measures and levels.

Figure 43. Separability indices of ‘helpen’ by harmonic mean of ranks for different measures and levels.

Visual comparison

In order to compare models along a defined set of parameters, the following groups of clouds were looked at side by side:

• Three groups of 8 models with BOUNDARIES:yes + FOC-POS:lex|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-WIN and BASE but also assess the effect of PPMI.
• Three groups of 8 models with FOC-WIN:5 + (LENGTH:5000 + SOC-POS:all) | (LENGTH:FOC + SOC-POS:nav), each with a different PPMI, in order to compare across different BOUNDARIES, FOC-POS and SOC-VECTOR but also assess the effect of PPMI.
• Three groups of 8 models with LEMMAPATH | LEMMAREL:selection + (LENGTH:5000 + SOC-POS:all) | (LENGTH:FOC + SOC-POS:nav), each with a different PPMI, in order to compare across different dependency-based templates and SOC-VECTOR but also assess the effect of PPMI.

In the first group, distances range up to 0.7 if PPMI:no and up to 0.6 otherwise; the highest values are between LEMMAREL and BOW-based models and between FOC-WIN:3 and all but FOC-WIN:5; the lowest are for LEMMAREL and LEMMAPATH pairs. In the second group, distances between models with different SOC-VECTOR lie between 0.35 and 0.5 if PPMI:weight, and slightly higher (up to 0.8) otherwise, where LENGTH:5000 + SOC-POS:all models with different FOC-POS are also far from each other; it’s much lower for those with the same SOC-VECTOR and much lower if the models share FOC-POS. The values are very similar if FOC-WIN is kept constant at another value. Finally, in the third group, the highest differences (up to 0.5 if PPMI:weight|selection, up to 0.7 otherwise) are between models with different SOC-VECTOR and lower if it’s the same, with the lowest values for LEMMAPATH pairs.

In the n-MDS clouds, there is a relatively good split between senses in all models, with contact between them but not much overlap. Between the BOW-based models, there seems to be less overlap in the BOUNDARIES:yes + LENGTH:FOC + SOC-POS:nav models. In the dependency-based, the senses are well split in LENGTH:FOC + SOC-POS:nav but overlap much more in LENGTH:5000 + SOC-POS:all.

In the t-SNE clouds, some clusters emerge for certain frequent lexical collocates, but the rest of the tokens become more disperse the higher the perplexity, hiding the clusters in perplexity 50. There are five main clusters: one of herroepen_1 gathered by ‘beslissing’, two of herroepen_2 gathered by ‘verklaring’ and ‘laat’ (“dit werd later herroepen”), and one dense herroepen_2 cluster gathered by ‘uitspraak’ that hangs out next to a less compact herroepen_1 cluster gathered by ‘worden’ and several legal terms (‘rechtbank’, ‘vonnis’, etc.).

In LENGTH:5000 + SOC-POS:all models, the ‘laat’ cluster is better shaped but the ‘uitspraak’ cluster loses its herroepen_1 friend and from perplexity 20 only the ‘beslissing’ cluster is clear, and well separated from everything else. All clusters are more clear in BOUNDARIES:yes + LENGTH:FOC + SOC-POS:nav.

Clustering and separability measures

One other way of selecting the “best” models, or of quantifying their quality, is to use separability indices (from the semvar package). The four main indices (“DR”, “SIL”, “SCP” with k=10, and “kNN” with k=10) have been computed and their values (global quality, quality by class and mean class quality) have been ranked. The 3 models whose values lie in the top 10% for all ranks are listed below.

• BOWbound10lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.50)
• LEMMAPATHselection2.PPMIweight.LENGTHFOC.SOCPOSnav (mds)
• LEMMAPATHselection3.PPMIweight.LENGTHFOC.SOCPOSnav (mds)

Of these methods, the ones with highest ranking v-measures across different methods are BOWbound10lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.50) and LEMMAPATHselection3.PPMIweight.LENGTHFOC.SOCPOSnav (mds), illustrated in Figure 47.

Figure 47. Best models of ‘herroepen’ according to separability indices and v-measures.

The actual separability indices for these two best models is shown in Table 9.

More on separability indices

While the “best” models were chosen based on a ranking, the range of the values is very different across lemmas. The curve of the different values by rank for herroepen is shown in Figure 48.

Figure 48. Separability indices of ‘herroepen’ by rank for different measures and levels.

Figure 49. Separability indices of ‘herroepen’ by harmonic mean of ranks for different measures and levels.

Visual comparison

In order to compare models along a defined set of parameters, the following groups of clouds were looked at side by side:

• Three groups of 9 models with BOUNDARIES:yes + FOC-WIN:3|10|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-POS and BASE but also assess the effect of PPMI.
• Three groups of 9 models with BOUNDARIES:yes + FOC-WIN:3|10|NA + LENGTH:5000 + SOC-POS:all, each with a different PPMI, in order to compare across different FOC-POS and BASE but also assess the effect of PPMI and SOC-VECTOR.
• One group of 9 models with (BOUNDARIES:yes + FOC-WIN:10 | LEMMAREL:group2) + LENGTH:FOC + SOC-POS:nav in order to compare across different FOC-POS, PPMI and BASE.
• One group of 9 models with LEMMAREL|LEMMAPATH + PPMI:weight|selection + LENGTH:FOC + SOC-POS:nav, in order to compare different dependency templates and PPMI.

In the first group, distances span between 0.1 and 0.6, with lowest values for dependency-based pairs of models, and with FOC-WIN making a greater difference than FOC-POS.

The nMDS solutions tend to exhibit areas dedicated to the different senses (unless FOC-WIN:3 + FOC-POS:lex), with various degrees of overlap, especially in the LENGTH:5000 + SOC-POS:all group.

In t-SNE solutions, perplexity 10 gives numerous tiny clusters where herhalen_3 (reflexive) starts to separate, but with no separability between the rest of the senses in LENGTH:5000 + SOC-POS:all; perplexities of 30 and 50 don’t offer much improvement on perplexity 20.

PPMI:weight models split herhalen_3 in two based on the occurrence of ‘geschiedenis’; the dependency based models, especially LEMMAREL:group2m are the ones that keep areas for herhalen_3 and herhalen_4 the best, although there is still some overlap between herhalen_1 and herhalen_2.

Clustering and separability measures

One other way of selecting the “best” models, or of quantifying their quality, is to use separability indices (from the semvar package). The four main indices (“DR”, “SIL”, “SCP” with k=10, and “kNN” with k=10) have been computed and their values (global quality, quality by class and mean class quality) have been ranked. The top 10% for harmonic mean of ranks includes 68 solutions; there is no overlap between the top 10% of the different measures -there is only overlap between the top 10% of all values of all individual measures except for “SIL”.

Of these methods, the ones with highest ranking v-measures across different methods are LEMMAREL2.PPMIselection.LENGTHFOC.SOCPOSnav (tsne.30) and LEMMAREL1.PPMIselection.LENGTHFOC.SOCPOSnav (tsne.20), illustrated in Figure 53.

Figure 53. Best models of ‘herhalen’ according to separability indices and v-measures.

The actual separability indices for these two best models is shown in Table 10.

More on separability indices

While the “best” models were chosen based on a ranking, the range of the values is very different across lemmas. The curve of the different values by rank for herhalen is shown in Figure 54.

Figure 54. Separability indices of ‘herhalen’ by rank for different measures and levels.

Figure 55. Separability indices of ‘herhalen’ by harmonic mean of ranks for different measures and levels.

Visual comparison

In order to compare models along a defined set of parameters, the following groups of clouds were looked at side by side:

• Three groups of 9 models with BOUNDARIES:yes + FOC-WIN:3|10|NA + LENGTH:5000 + SOC-POS:all``, each with a differentPPMI, in order to compare across differentFOC-POSandBASEbut also assess the effect ofPPMI`.
• Three groups of 9 models with BOUNDARIES:yes + FOC-WIN:3|10|NA + LENGTH:5000 + SOC-POS:all, each with a different PPMI, in order to compare across different FOC-POS and BASE but also assess the effect of PPMI and SOC-VECTOR.
• One group of 9 models with (BOUNDARIES:yes + FOC-WIN:10 | LEMMAREL:group2) + LENGTH:FOC + SOC-POS:nav in order to compare across different FOC-POS, PPMI and BASE.
• One group of 9 models with LEMMAREL|LEMMAPATH + PPMI:weight|selection + LENGTH:FOC + SOC-POS:nav, in order to compare different dependency templates and PPMI.

Distances span between 0.1 for LENGTH:5000 + SOC-POS:all models and reach 0.8 with LENGTH:5000 + SOC-POS:all; they are lowest between LEMMAPATH pairs and highest between pairs with some LEMMAREL model.

In nMDS solutions, (LEMMAPATH | FOC-POS:lex) + PPMI:selection|weight + LENGTH:FOC + SOC-POS:nav show two main cores, one for each sense, with a respectable separation between them, while there is one main cloud in the rest of the models, showing some overlap in LEMMAREL and LENGTH:5000 + SOC-POS:all.

In t-SNE models, the sizes of the clusters varies across models and perplexity values, but both senses remain always well separated (unless PPMI:no + LENGTH:5000 + SOC-POS:all). huldigen_1 (“honor someone”) is always much more widespread than huldigen_2 (“hold a point of view”); the structure seems most clear with perplexity 20, but it spreads much wider with higher values.

In terms of pockets, huldigen_2 presents some strong, compact clusters based on the occurrence of ‘principe’ (which can be seen in some PPMI:weight models in nMDS solutions) and, if PPMI:weight, ‘standpunt’ and ‘opvatting’. huldigen_1 mostly exhibits a strong core in FOC-POS:lex and sometimes in LEMMAPATH | LEMMAREL:group2, marked by the presence of ‘worden’. In dependency-based and FOC-WIN:10 models, if PPMI:weight + LENGTH:FOC + SOC-POS:nav, there is also a ‘kampioen’ cluster to be found for huldigen_1, but it fades away with larger perplexities.

Clustering and separability measures

One other way of selecting the “best” models, or of quantifying their quality, is to use separability indices (from the semvar package). The four main indices (“DR”, “SIL”, “SCP” with k=10, and “kNN” with k=10) have been computed and their values (global quality, quality by class and mean class quality) have been ranked. The top 10% for all ranks includes 18 solutions:

• BOWnobound5lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.30)
• BOWnobound5lex.PPMIweight.LENGTHFOC.SOCPOSall (tsne.30)
• BOWbound5lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.50)
• BOWbound5lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.30)
• BOWbound10lex.PPMIweight.LENGTHFOC.SOCPOSall (tsne.50)
• BOWbound10lex.PPMIweight.LENGTHFOC.SOCPOSall (tsne.30)
• BOWnobound5lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.20)
• BOWnobound5lex.PPMIweight.LENGTHFOC.SOCPOSall (tsne.50)
• BOWnobound5lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.50)
• BOWbound10lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.20)
• BOWbound10lex.PPMIweight.LENGTHFOC.SOCPOSall (tsne.20)
• BOWbound10lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.50)
• BOWnobound10lex.PPMIweight.LENGTHFOC.SOCPOSall (tsne.20)
• BOWbound5lex.PPMIselection.LENGTHFOC.SOCPOSall (tsne.30)
• BOWnobound10lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.30)
• BOWnobound10lex.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.20)
• BOWbound5lex.PPMIselection.LENGTHFOC.SOCPOSall (tsne.50)
• BOWbound10lex.PPMIweight.LENGTHFOC.SOCPOSall (tsne.10)

Of these methods, the ones with highest ranking v-measures across different methods are BOWnobound10lex.PPMIweight.LENGTHFOC.SOCPOSall (tsne.20) and BOWbound10lex.PPMIweight.LENGTHFOC.SOCPOSall (tsne.10), illustrated in Figure 59.

Figure 59. Best models of ‘huldigen’ according to separability indices and v-measures.

The actual separability indices for these two best models is shown in Table 11.

More on separability indices

While the “best” models were chosen based on a ranking, the range of the values is very different across lemmas. The curve of the different values by rank for huldigen is shown in Figure 60.

Figure 60. Separability indices of ‘huldigen’ by rank for different measures and levels.

Figure 61. Separability indices of ‘huldigen’ by harmonic mean of ranks for different measures and levels.

Visual comparison

In order to compare models along a defined set of parameters, the following groups of clouds were looked at side by side:

• Three groups of 9 models with BOUNDARIES:yes + FOC-WIN:3|10|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-POS and BASE but also assess the effect of PPMI.

Distances span between 0.4 and 0.8 for pairs with some BOW-based model, but lower for those with same FOC-WIN, as well as for LEMMAPATH pairs.

In nMDS solutions, clouds mostly have a dense core with a halo and a lot of overlap between the senses. The t-SNE solutions, especially with perplexity 20, do show three main clusters –and sometimes a couple more– but they don’t really match the senses; higher perplexities are just more disperse versions of the same topology.

The most clear small clusters are one with ‘schuld’ as co-occurrence (mostly herstructureren_1) in PPMI:weight|selection, one with ‘grondig’ in PPMI:weight and one with aan in most FOC-WIN:all | LEMMAPATH models. The most interesting organization, however, is the three main clusters, that take up a triangular formation in LEMMAREL:group2 and a line in LEMMAPATH. The groups seem to be characterized by the co-occurrence of ‘om’+‘te’, ‘worden’ and ‘worden’+‘om’+‘bedrijf’. Rather than exhibiting a predominant sense, they seem to repel senses: ‘om’+‘te’ seems to repel herstructureren_2, ‘worden’ herstructureren_3, and ‘worden’+‘om’+‘bedrijf’, herstructureren_1 (fascinating, right?).

Clustering and separability measures

One other way of selecting the “best” models, or of quantifying their quality, is to use separability indices (from the semvar package). The four main indices (“DR”, “SIL”, “SCP” with k=10, and “kNN” with k=10) have been computed and their values (global quality, quality by class and mean class quality) have been ranked. The top 10% for harmonic mean of ranks includes only 15 solutions; there is no overlap between the top 10% of the different measures –actually, there is only overlap between the top 10% of all values of “kNN” and “SCP”.

• BOWnobound10all.PPMIweight.LENGTHFOC.SOCPOSall (tsne.20)
• BOWnobound10all.PPMIweight.LENGTHFOC.SOCPOSnav (tsne.20)
• LEMMAPATHselection2.PPMIselection.LENGTHFOC.SOCPOSall (mds)
• LEMMAPATHweight.PPMIselection.LENGTH5000.SOCPOSnav (tsne.30)
• LEMMAPATHweight.PPMIselection.LENGTH5000.SOCPOSnav (tsne.10)
• LEMMAPATHweight.PPMIselection.LENGTH5000.SOCPOSnav (tsne.50)
• LEMMAPATHselection2.PPMIno.LENGTHFOC.SOCPOSnav (mds)
• LEMMAPATHselection2.PPMIselection.LENGTHFOC.SOCPOSnav (mds)
• LEMMAPATHselection2.PPMIno.LENGTHFOC.SOCPOSall (mds)
• LEMMAPATHselection2.PPMIweight.LENGTHFOC.SOCPOSall (mds)
• LEMMAPATHweight.PPMIselection.LENGTH5000.SOCPOSnav (tsne.20)
• LEMMAREL2.PPMIselection.LENGTHFOC.SOCPOSnav (mds)
• LEMMAREL2.PPMIselection.LENGTHFOC.SOCPOSall (tsne.50)
• LEMMAREL2.PPMIno.LENGTHFOC.SOCPOSnav (mds)
• LEMMAPATHselection2.PPMIweight.LENGTHFOC.SOCPOSnav (mds)

Of these methods, the ones with highest ranking v-measures across different methods are LEMMAREL2.PPMIselection.LENGTHFOC.SOCPOSall (tsne.50) and LEMMAPATHweight.PPMIselection.LENGTH5000.SOCPOSnav (tsne.50), illustrated in Figure 65.

Figure 65. Best models of ‘herstructureren’ according to separability indices and v-measures.

The actual separability indices for these two best models is shown in Table 12.

More on separability indices

While the “best” models were chosen based on a ranking, the range of the values is very different across lemmas. The curve of the different values by rank for herstructureren is shown in Figure 66.

Figure 66. Separability indices of ‘herstructureren’ by rank for different measures and levels.

Figure 67. Separability indices of ‘herstructureren’ by harmonic mean of ranks for different measures and levels.

Visual comparison

In order to compare models along a defined set of parameters, the following groups of clouds were looked at side by side:

• Three groups of 9 models with BOUNDARIES:yes + FOC-WIN:3|10|NA + LENGTH:FOC + SOC-POS:nav, each with a different PPMI, in order to compare across different FOC-POS and BASE but also assess the effect of PPMI.
• Three groups of 8 models with (BOUNDARIES:yes + FOC-WIN:10 | LEMMAPATH) as first order alternatives and (LENGTH:5000 + SOC-POS:all) | (LENGTH:FOC + SOC-POS:nav) as second order alternatives, each with a different PPMI, in order to compare across different FOC-POS and SOC-VECTOR but also assess the effect of PPMI.

Distances span between 0.1 and 0.7, highest for pairs with LEMMAREL1 and lowest for pairs with PATH; distances between LEMMAREL models decrease with PPMI:no, while distances between pairs with one FOC-POS:lex increase. In the second group, distances between models with different SOC-VECTOR or LENGTH:5000 + SOC-POS:all + PPMI:selectoin|2no are around 0.5 or higher, while the rest (and particularly, all LEMMAPATH pairs) are lower: SOC-VECTOR can make a greater difference than BASE.

The LEMMAREL:group1 models perform consistently worse than the others, with much more overlap and less clear clusters, so it will be excluded from the description below.

In the nMDS solutions ther eis a clear cluster of tokens joined by the ocurrence of ‘in’+‘ere’ (herstellen_2) and a couple of outliers that can really distort the picture, especially in PPMI:selection and/or LENGTH:5000 + SOC-POS:all (the most powerful one only has one context word in those models, namely ‘naaien’). The infrequent sense herstellen_6 (intransitive, for financial entities) always overlaps with or is included within herstellen_4 (reflexive; often with “economie” or similar entities as subject), so they will be spoken of as one. If PPMI:weight and/or LENGTH:FOC + SOC-POS:nav, all senses except herstellen_3 (transitive, referring to mistakes; quite infrequent) are relatively well separated from each other, taking different areas in the cloud, and the ‘in ere’ cluster hovers close to the major herstellen_2 group.

In t-SNE solutions, perplexity 20 strengthens the organization in PPMI:weight models, but does not improve the rest that much, and higher perplexities only increase dispersion. The ‘in ere’ cluster is always well defined, but not necessarily in the vicinity of herstellen_2 (that may be random, although it does occur more often with perplexity 20 than with 10). PPMI:weight models add a furhter cluster marked by the occurrence of ‘evenwicht’ (also herstellen_2), best separated if FOC-WIN:10 + FOC-POS:all | LEMMAPATH:selection3, while PPMI:selection|no clusters most of herstellen_5 by the co-occurrence of ‘van’+‘zijn’ in the same models.

Other than these clusters, the rest of the tokens form a wider mass, with the main senses taking up specific areas. This may allow for a meaningful organization, such as the herstellen_4 group between the transitive senses on one side and herstellen_5 (intransitive, “to heal”) on the other in (FOC-WIN:10 + FOC-POS:all | LEMMAPATH:selection3) + PPMI:weight with perplexity 20.

Clustering and separability measures

One other way of selecting the “best” models, or of quantifying their quality, is to use separability indices (from the semvar package). The four main indices (“DR”, “SIL”, “SCP” with k=10, and “kNN” with k=10) have been computed and their values (global quality, quality by class and mean class quality) have been ranked. The top 10% for harmonic mean of ranks includes 46 solutions; there is no overlap between the top 10% of the different measures –actually, there is only overlap between the top 10% of all values of “kNN” and “SCP”.

Of these methods, the ones with highest ranking v-measures across different methods are LEMMAPATHselection3.PPMIweight.LENGTH5000.SOCPOSnav (tsne.10) and BOWbound10all.PPMIweight.LENGTH5000.SOCPOSnav (tsne.20), illustrated in Figure 71.

Figure 71. Best models of ‘herstellen’ according to separability indices and v-measures.

The actual separability indices for these two best models is shown in Table 13.

More on separability indices

While the “best” models were chosen based on a ranking, the range of the values is very different across lemmas. The curve of the different values by rank for herstellen is shown in Figure 72. DR values were ignored for 2 solutions because they were too high –larger than 10– in some levels (globqual, meanclassqual, classqual_herstellen_2, classqual_herstellen_3, classqual_herstellen_4, classqual_herstellen_5), up to 682.1.

Figure 72. Separability indices of ‘herstellen’ by rank for different measures and levels.

Figure 73. Separability indices of ‘herstellen’ by harmonic mean of ranks for different measures and levels.